A distributed storage system includes multiple sites which are in communication via a network. A separate copy of the same set of data may be maintained at each one of the sites. In one implementation of such a system, the data set represents a logical unit of storage (LUN) which can be presented to clients at the different sites. A client associated with a particular site may access the virtualized LUN by performing IOs on data maintained at the associated site. Changes made to data at the associated site are communicated to other sites to prompt corresponding changes which help maintain consistency of the copies of the data set across sites. Procedures may be used to prevent contemporaneous changes by different clients. For example, a portion of data such as a range of blocks can be locked down across sites so that two clients are prevented from contemporaneously modifying that data in a manner which creates inconsistency.
A problem can arise when there is a loss of communication between sites. Loss of communication can occur for various reasons, including but not limited to physical breakage of communication media (e.g., accidentally cutting a fiber), node failure, extreme weather, routine maintenance, and/or intentional scheduling of communication availability. Although loss of communication is usually temporary, there is a possibility that two or more clients at different sites could modify the same portion of data while procedures for maintaining consistency are rendered inoperative by the communication loss. For example, if two different sites were to write to the virtualized LUN then the two sites might write incompatible data. Similarly if one site were to write to the virtualized LUN and a different site were to read that data it could read stale data. Consequently, criteria such as a “bias” setting or a “vote” provided by a third “witness” site may be used to limit access to the LUN while inter-site communication is not possible. In particular, during communication breakage, only one site is permitted to access the LUN. Consequently, clients at other sites cannot read or write to the LUN. This helps ensure that data consistency is maintained, but may inhibit client progress at sites which are not provided access to the LUN.
Similarly, when there are three or more sites that are sharing the same set of data, and communication failures occur resulting in a partition of the sites, then various bias, voting, cluster witness or other quorum rules can be used to give write or read access to a subset of nodes that are in communication with each other and to prohibit write or read access from other nodes and thereby maintain data consistency until such time as communication is restored and all copies of the sets of data can be made consistent with each other.